Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TPV: remove singularity no-mount tmp flag for funannotate_predict #1302

Merged

Conversation

sanjaysrikakulam
Copy link
Member

@sanjaysrikakulam sanjaysrikakulam commented Aug 21, 2024

The --no-mount tmp is set default by Galaxy; this flag allows specific system mounts to be disabled.

We override it in this PR (org had issues in the past because of this default behavior, which was addressed through this PR) so this tool can use the host's /tmp file system that is by default bind mounted into the container by singularity.

For reference, Org uses this in their singularity destinations. However, we are overwriting it only for this tool instead of setting it globally.

This fixes the write permission and NFS resource busy errors we see in this tool's user error reports.

Tested:

  1. I imported a user history where this tool failed
  2. Reran the job (the job failed)
  3. Edited the job script (galaxy_72623614.sh) where I removed the --no-mount tmp and submitted the job using the below job file via condor_submit
Universe = vanilla
Executable = /data/jwd02f/main/072/623/72623614/galaxy_72623614.sh
Log = /data/jwd02f/main/072/623/72623614/galaxy_72623614.condor.log
Request_cpus = 12
Request_memory = 12288
Queue

The job is still running (it has been 30 minutes; probably not a simple job); usually, the job fails in the 1st few minutes with either the NFS resource busy or a write permission error.

this is set default by Galaxy. We override it here so this job can use the host's /tmp file system bind mounted into the container. This fixes the write permission error as well as the NFS resource busy error
@@ -1338,6 +1338,7 @@ tools:
_GALAXY_JOB_TMP_DIR: '/tmp'
params:
singularity_run_extra_arguments: "--env GENEMARK_PATH=/usr/local/tools/genemark/etp.for_braker/bin/gmes/"
singularity_no_mount: null
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

null and not false?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This variable seems to take a list of strings, and the org defines it as null to overwrite the defaults (\tmp). Unfortunately, there is no reference to this variable anywhere in the code base, and it lacks documentation. So, I think it will just add --no-mount null in the final populated singularity command. Let's try it out and see what happens.

@bgruening
Copy link
Member

Pretty cool, lets hope this will work finally!

@bgruening bgruening merged commit 35788cc into usegalaxy-eu:master Aug 22, 2024
4 checks passed
@bgruening
Copy link
Member

Feel free to redeploy...

@sanjaysrikakulam
Copy link
Member Author

Update: I was able to successfully run the funannotate_predict following this tutorial. This issue is finally solved.

BTW: Setting the params singularity_no_mount: null the --no-mount /tmp is not added to the populated singularity command in the galaxy job script.

@bgruening
Copy link
Member

Awesome, thanks a lot!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants